A stochastic subspace approach to gradient-free optimization in high dimensions

We present a stochastic descent algorithm for unconstrained optimization that is particularly efficient when the objective function slow to evaluate and gradients are not easily obtained, as in some PDE-constrained machine learning problems. The maps gradient onto low-dimensional random subspace of dimension $$\ell$$ at each iteration, similar coordinate but without restricting directional derivatives be along axes. Without requiring full gradient, this mapping can performed by computing (e.g., via forward-mode automatic differentiation). give proofs convergence expectation under various convexity assumptions well probabilistic results strong-convexity. Our method provides novel extension well-known Gaussian smoothing technique subspaces greater than one, opening doors new analysis more one derivative used iteration. also provide finite-dimensional variant special case Johnson–Lindenstrauss lemma. Experimentally, we show our compares favorably descent, smoothing, BFGS (when calculated differentiation) on problems from shape literature.